Make MonitorJobPoller populate thread context for downstream request interception by toepkerd · Pull Request #2107 · opensearch-project/alerting

toepkerd · 2026-04-28T01:56:05Z

Description

Job poller now populates the thread context with information about the external data source so downstream flows can properly intercept it and make Search and PPL calls to external data source.

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…asis interception Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

riysaxen-amzn · 2026-04-28T16:38:36Z

        const val POLLER_THREAD_COUNT = 10
        const val POLL_INTERVAL_MS = 1000L
+
+        // thread context header keys for request interception


nit: I think we should follow a consistent naming convention for all these headers but needs to be discussed this across team

These constants are actually existing header names from other code owners that we are reusing. The only new and original const here is IS_BACKGROUND_JOB_HEADER = "alerting-is-background-job"

engechas · 2026-04-28T16:42:34Z

+        threadContext.putHeader(IS_BACKGROUND_JOB_HEADER, "true")
+
+        // TODO: in long term, may need to generalize to aos data source type
+        threadContext.putHeader(SERVICE_NAME_HEADER, "aoss")


Is this information in the monitor config already? If so we can pull it from there instead of hardcoding

I don't think it's anywhere in the Monitor config, but there is a REMOTE_METADATA_SERVICE_NAME setting I could retrieve the value from. Will explore reading from this value instead of hard-coding aoss.

If reading from REMOTE_METADATA_SERVICE_NAME approach is used, we must be mindful of conventions for naming services. For collection data sources, I think we can trust the service name will always be aoss.

In the long term, if we support domain data sources, we will need to match existing service name conventions in the interception logic. Whether they favor aos or es as the service name, our logic for populating these remote metadata settings (outside the scope of this PR) will need to be ready to match that convention if reading from settings is to work.

monitor.target.type will have this header info

https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Target.kt

@riysaxen-amzn @MohammedAghil can help ensure that this header's value will match exactly what the header needs or what a setting can hold as a kv map where key will be target.type and value would be header value

I see, I misunderstood the monitor.target.type field, thanks for clarifying. I will double check the values that this field can take on (other than the default value of "local"). I will add fail-fast validations for the type.

eirsep · 2026-04-28T22:00:31Z

+            )
+        }
+
+        if (region.isBlank()) {


this should be a check before poller threads are bootstrapped itself near
val provider = requireNotNull(accountIdProvider) { "accountIdProvider must be set before starting" }
val sqs = requireNotNull(sqsClient) { "sqsClient must be set before starting" }

eirsep · 2026-04-28T22:03:04Z

+        threadContext.putHeader(IS_BACKGROUND_JOB_HEADER, "true")
+
+        // TODO: in long term, may need to generalize to aos data source type
+        threadContext.putHeader(SERVICE_NAME_HEADER, "aoss")


monitor.target.type will have this header info

https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Target.kt

eirsep · 2026-04-28T22:04:31Z

+        threadContext.putHeader(IS_BACKGROUND_JOB_HEADER, "true")
+
+        // TODO: in long term, may need to generalize to aos data source type
+        threadContext.putHeader(SERVICE_NAME_HEADER, "aoss")


@riysaxen-amzn @MohammedAghil can help ensure that this header's value will match exactly what the header needs or what a setting can hold as a kv map where key will be target.type and value would be header value

eirsep · 2026-04-28T22:22:13Z

+        threadContext.putHeader(SERVICE_NAME_HEADER, "aoss")
+
+        // external customer data source endpoint, to run search/ppl against
+        threadContext.putHeader(OPENSEARCH_ENDPOINT_HEADER, monitor.target!!.endpoint)


plz check that it's not null/empty before you set it.

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

eirsep · 2026-04-29T20:55:02Z

            Setting.Property.NodeScope, Setting.Property.Dynamic
        )
+
+        val JOB_QUEUE_TARGET_TYPE_TO_SERVICE_NAME = Setting.groupSetting(


will this default to empty map

Yes.

If no settings with this prefix is configured anywhere, MonitorJobPoller construction in AlertingPlugin.kt effectively gets an empty keySet() (like get(SETTINGS.Empty)). This means every Monitor Job Poller job will fail at thread context writing with "invalid target type" exception. This is intentional behavior: values for these settings must be configured in the following way in opensearch.yml:

plugins.alerting.monitor.target_type_to_service_name.target_type_1: service_name_1 plugins.alerting.monitor.target_type_to_service_name.target_type_2: service_name_2

eirsep · 2026-04-29T20:59:57Z

        )
+
+        val JOB_QUEUE_TARGET_TYPE_TO_SERVICE_NAME = Setting.groupSetting(
+            "plugins.alerting.external_scheduler.type_to_service.",


can this be renamed to plugins.alerting.monitor.target_type_to_service_name.

eirsep

minor comments but rest LGTM

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

Making job poller populate thread context with data source info for O…

383340a

…asis interception Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

toepkerd force-pushed the main branch from e67a388 to 1e553ce Compare April 28, 2026 01:56

added unit test

ac44c34

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

toepkerd force-pushed the main branch from 1e553ce to ac44c34 Compare April 28, 2026 03:07

toepkerd marked this pull request as ready for review April 28, 2026 16:25

toepkerd requested review from AWSHurneyt, KashKondaka, amsiglan, bowenlan-amzn, eirsep, engechas, getsaurabh02, goyamegh, lezzago, manaswini1920, nishtham-amazon, rishabhmaurya, riysaxen-amzn and vikhy-aws as code owners April 28, 2026 16:25

toepkerd changed the title ~~Make MonitorJobPoller populate thread context for Oasis interception~~ Make MonitorJobPoller populate thread context for downstream request interception Apr 28, 2026

adjusting comments

2a71174

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

riysaxen-amzn reviewed Apr 28, 2026

View reviewed changes

Comment thread alerting/src/main/kotlin/org/opensearch/alerting/service/MonitorJobPoller.kt Outdated

riysaxen-amzn reviewed Apr 28, 2026

View reviewed changes

engechas reviewed Apr 28, 2026

View reviewed changes

eirsep requested changes Apr 28, 2026

View reviewed changes

eirsep reviewed Apr 28, 2026

View reviewed changes

toepkerd-zz added 4 commits April 28, 2026 16:44

added extra validations and target -> service name mappings

c7ba88f

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

putting mappings behind consts

5eff8a2

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

fixing linter issues

b779d5c

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

passing Target directly to helper instead of full Monitor object

e109e17

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

adding operation type header and extra tests

0c9eed8

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

eirsep reviewed Apr 29, 2026

View reviewed changes

Comment thread alerting/src/main/kotlin/org/opensearch/alerting/service/MonitorJobPoller.kt

toepkerd-zz added 2 commits April 28, 2026 20:50

changing name of is background job flag thread context header

7dcdcfe

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

putting target type -> service name mappings behind setting

102583b

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

eirsep reviewed Apr 29, 2026

View reviewed changes

eirsep approved these changes Apr 29, 2026

View reviewed changes

renaming target type to service name mappings setting

bf6feb0

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

riysaxen-amzn approved these changes Apr 29, 2026

View reviewed changes

toepkerd merged commit 642b702 into opensearch-project:main Apr 29, 2026
21 of 23 checks passed

Conversation

toepkerd commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Check List

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toepkerd Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toepkerd Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eirsep Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eirsep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

toepkerd commented Apr 28, 2026 •

edited

Loading

toepkerd Apr 28, 2026 •

edited

Loading

toepkerd Apr 29, 2026 •

edited

Loading

eirsep Apr 29, 2026 •

edited

Loading